A focused approach to assessing fidelity 1 A focused approach to assessing prevention program fidelity

The primary goals of the present study were 1) to explore the types and frequencies of adaptation reported by facilitators; 2) to document their reasons for making those adaptations; and 3) to propose an efficient approach to the study of prevention program fidelity based on interview data. We interviewed 42 program facilitators involved in a large-scale dissemination about their implementation of a community-based prevention program. Interview questions addressed facilitators’ attitudes about program fidelity and the various types of changes, deletions, and additions they made. Although nearly all facilitators reported that fidelity to program curriculum was important, most also reported adapting the program. The most frequent reason facilitators gave for adaptation was deleting or changing material because they ran out of time. We employed a method used in business and health care settings (the Pareto principle, or the “law of the vital few”) to identify the minority (30%) of types of adaptation that accounted for a majority (70%) of all adaptations reported. Similarly, we identified the minority of reasons (25%) for adaptation that accounted for a majority (75%) of reasons reported. This approach helps to focus the assessment of fidelity on those adaptations that occur with high frequency, since low-frequency events are unlikely to have a significant impact on large-scale outcomes. High-frequency adaptations can then be targeted to determine their effect on outcomes; to address in training; and to assess on an ongoing basis for continuous quality improvement. A focused approach to assessing fidelity 3 The goal of the present paper is to present a focused approach to the study of program fidelity in community-based programs. First, we present details of interviews with facilitators of an evidence-based program with a statewide dissemination. The program (Strengthening Families Program for Parents and Youth 10-14) is standardized through use of videos and manuals with scripted activities. Interviews were designed to explore the range of adaptations made by facilitators when conducting the program, as well as their reasons for making them. Second, we examine the frequencies of those adaptations and reasons for making them. Finally, we explore the utility of an approach used in healthcare and business to focus on high-frequency events for continuous quality improvement. Studying Fidelity The term “intervention fidelity” refers to the match between an intervention as it was intended to be delivered and the intervention as it is actually delivered in real-world circumstances (Chen, 1990; Heflinger, 1996; Mowbray, Holter, Teague & Bybee, 2003). Fidelity is a broad term and may encompass aspects of programming other than content or mode of delivery. The content of most evidence-based programs is developed using explicit theory that has been tested and refined over the course of multiple studies, and the theory underlying program content links program implementation to program outcomes – that is, developers include specific content because they believe it will result in specific outcomes. Thus, deviations from fidelity to program content may have unintended effects on outcomes. Structural or service-delivery aspects of the program may also be important, and even when a theory of program structure is not specified, it is often implicit (Chen, 1990). Replicated, randomized, clinical trials (RCTs) are able to restrict variation in contextual variables (e.g. A focused approach to assessing fidelity 4 program location, facilitator education, or group size). In real-world implementation, the implicit theory underlying these structural details may be violated. For example, a developer may create a program intended to be delivered over ten sessions. Program attendees, however, may come to fewer sessions and thus do not receive the intended “dosage” of program content, which may diminish program effectiveness. Similarly, there are theoretical and empirical reasons to believe that group size, for example, may be an important determinant of participant learning and behavior change (Marsden & Mathayalakan, 1999); effects on program outcomes of variability in this and many other contextual variables remain unexplored. Finally, a program may be delivered to populations other than those intended, which may in turn have multiple effects on program fidelity. If a program intended for young children is presented to older children, the link between program content (developed for a particular developmental stage) and program effectiveness is likely to be affected. Program facilitators may also adapt content more, sometimes out of necessity, when their program population is very different from the intended population: language and family structure, for example, may force adaptation of content or structure or both. The degree to which outcomes are affected by deviations from fidelity is a significant concern in the translation of prevention programs from research-based efficacy trials to community-based implementation (Dusenbury, Brannigan, Falco & Hansen, 2003; Greenberg, Domitrovich & Bumbarger, 2001; Kaftarian, Robinson, Compton, Davis, & Volkow, 2004). Increasingly granting agencies at state and federal levels are requiring that programs chosen for funding be designated as “model” or “best practice” programs, whose efficacy has been demonstrated through RCTs. These designations reflect a determination that program activities are causally related to desired outcomes. However, research-driven trials are more highly A focused approach to assessing fidelity 5 controlled than community-based programs. When programs move into their intended destination, the “real world”, substantial adaptation to program materials may occur. Thus, an important reason to study fidelity is to determine whether specific adaptations are associated with program impact (Botvin & Griffin, 2005). A second reason is to improve implementation on a continuous basis (Limber, 2006). Program fidelity is difficult to assess on an ongoing basis, especially in multisite programs disseminated across large areas, and especially when the programs are communitydriven (Ogden, Forgatch, Askeland, Patterson, & Bullock, 2005). One barrier to effective assessment of fidelity is that the universe of potential deviations from fidelity to structure and content is large, and we do not know 1) how many of these deviations occur on a regular basis; and 2) which are most important in terms of outcome. Thus, it is difficult to know what to assess. There are also practical difficulties in large-scale assessment of fidelity, especially without a targeted focus on potentially influential deviations (Limber, 2006). One method might be to require facilitators to complete implementation surveys after each instance of a program. However, some research has shown that self ratings of fidelity may not be as reliable as observer ratings and that observer ratings are more closely associated with program outcomes (Lillehoj, Griffin, & Spoth, 2004; Spoth, Guyll, & Goldberg-Lillehoj, 2002). This suggests that observation would be a preferable method of assessment, but large-scale observation is not feasible on an ongoing basis. A more reliable and efficient method of ongoing assessment of fidelity, though, is necessary to provide feedback for continuous quality improvement to all program implementers. Understanding Adaptation A focused approach to assessing fidelity 6 Systematic data describing why facilitators change programs are lacking. There are a number of studies on adaptation for cultural reasons (Castro, Barrera, & Martinez, 2004; Kumpfer, Alvarado, Smith & Bellamy, 2002), and facilitator attributes may also be associated with adaptation (Machin & Fogarty, 2003; St. Pierre & Kaltreider, 2004). Some researchers have explored attributes of teachers who adapt school-based prevention programs (Ringwalt, Ennis, Vincus, & Simons-Rudolph, 2004; Rohrbach, Graham, & Hansen, 1993), but it is likely that community-based facilitators comprise a more heterogeneous group, and the context in which they deliver programs is also different from that of classroom-based programs (Castro, Barrera, & Martinez, 2004). Ringwalt and colleagues conducted a survey of 1905 public school teachers of substance abuse curricula and found that beliefs about program effectiveness and organizational support were among predictors of fidelity in adaptation (Ringwalt et al., 2003), and that group attributes (e.g. students’ special needs) predicted adaptation (Ringwalt et al., 2004). However, there is a dearth of research asking facilitators to describe the range of adaptations they implement and their reasons for making them. This knowledge is important because it can inform approaches to preventing or minimizing adaptations during training, technical assistance, or quality assessment. Focusing the Study of Fidelity: The Pareto Principle The Pareto principle, also known as the “law of the vital few”, is based on the observation that most effects are determined by a small number of causes. The principle originated in the study of political economics in the middle of the last century, when Italian economist Vilfredo Pareto noted that 80% of the nation’s income was distributed to 20% of the population. Since that time, the principle has been applied extensively to problems in the academic fields of economics and engineering (c.f. Shoham, 1999), to industry and business A focused approach to assessing fidelity 7 management (Juran, 1954), to stock investment and everyday life (Koch, 1999), and across multiple fields as a quality improvement tool (Gryna, Chua, & DeFeo, 2005). For example, the Institute for Healthcare Improvement used data from hospitals nationwide to determine that changing 6 basic hospital practices would prevent 100,000 iatrogenic deaths annually (Berwick, 1995). If the Pareto principle holds true in the implementation of prevention programs, we would expect to find that a few types of adaptation, out of many possible types, account for a majority of all adaptations that occur. Similarly, only a few reasons for adaptation, out of many possible reasons, would account for most of why facilitators deviate from strict adherence to a standardized program. This information would be helpful in several ways. First, it would narrow the need to assess effects of adaptation on program outcomes, since very low-frequency events are unlikely to have a significant effect on large-scale program outcomes. Second, it would enable program developers to focus on preventing those few types by addressing the “vital” few reasons why they occur. Third, it would allow for more efficient efforts to conduct ongoing assessment of fidelity for purposes of quality improvement. In order to examine whether this is a feasible approach, two steps are necessary: first, the range of potential adaptations, or types of adaptations, must be defined. Second, the frequencies of those adaptations must be explored. Ultimately it will be important to determine general trends that cut across individual prevention programs and types of programs -in other words, to determine whether certain generic types of adaptation occur more frequently in implementation of school-based and family-based programs. As a first step in this direction, however, the present study explores these questions in the context of a statewide dissemination of a single evidence-based program. Background of the Present Study A focused approach to assessing fidelity 8 In 2000, Extension faculty from Washington State University began a coordinated dissemination of a well-known substance abuse prevention program, the Strengthening Families Program for Parents and Youth 10-14 (SFP) (Kumpfer, Molgaard, and Spoth, 1996). SFP is a universal substance abuse prevention program that has been demonstrated efficacious in numerous studies (Foxcroft, Ireland, Lister-Sharp, Lowe & Breen, 2003; Spoth, Redmond, & Shin, 2001). The program is delivered one evening a week for seven weeks. During the first hour of each session, children and parents receive skills training separately. During the second hour, they come together to participate in family skills training. Parent, child and family risk factors are addressed through interactive exercises. The material is covered through activities, lectures, videos, discussions, and role-plays, and homework is assigned at the end of each session (Molgaard, Kumpfer, & Fleming, n.d.). Extension faculty trained by the program developer regularly conduct standardized trainings for program facilitators and coordinate local series of implementations throughout the state; most of these facilitators conduct an outcome evaluation and submit implementation reports. The present study was designed as an initial attempt to focus our efforts to create an efficient and practical ongoing fidelity assessment to provide feedback to program implementers. Study Questions and Hypotheses We conducted interviews of facilitators statewide, asking what types of adaptations they made on a regular basis: Do they add material to the standardized curriculum, delete program components, or change program content? Which components were they most likely to adapt? Second, what are the reasons for the adaptations they make: Do they most often adapt because they disagree with program content, because they feel their personal expertise has something to add to the program, or because of specific attributes of their clientele? And third, are some A focused approach to assessing fidelity 9 facilitators more likely to adapt program material than others? For example, are more experienced facilitators less likely to adapt? Do age and education play a role in willingness to adapt? Do attitudes about program fidelity and beliefs about the program relate to reported adaptation practices? We hypothesized that a minority of types of adaptation would account for a majority of adaptations reported. Similarly, we hypothesized that a minority of types of reasons for adaptation would account for a majority of reasons reported. It should be emphasized that because the study examines facilitator adaptations of a single program, the types of adaptation likely do not represent the full range of possible adaptations to evidence-based programs. However, we believe that the approach we use to assess fidelity can easily be applied to any community-based prevention program. Method Sample Interview sample. From fall of 2000 through summer 2004, WSU Extension faculty trained 301 SFP facilitators, of whom 65 had implemented SFP at least one time by spring 2005. We were able to contact 52 of the 65 facilitators through email or telephone, 51 of whom indicated willingness to participate in the study. Forty-one (80%) of the 51 returned consent forms and completed the interview process. Facilitator occupations included parent volunteers (n = 3); school teachers, school personnel and administrators (n = 13); non-school counselors (n = 5); justice system personnel (n = 1); representatives from state and local social service agencies (n = 11); and those who identified themselves as prevention specialists (n = 8). A majority of providers were female (n = 30; 73.2%). Facilitator ethnicity was mainly European American (n = 31; 75.6%), with A focused approach to assessing fidelity 10 facilitators of Latino descent being the second most common ethnicity (n = 5; 12.2%). Average age of facilitators in the study was 48.5 years. Facilitators reported high levels of experience with interactive teaching methods (mean of 5.6 on a scale of 1 to 7, with 1 representing “Absolutely no previous experience” and 7 representing “A lot of previous experience”). Also, many facilitators had implemented SFP several times, and many had implemented a variety of other evidence-based programs as well. Measures Facilitator interview. The interview developed for the study consisted of questions in six categories: 1) demographic and employment information; 2) training and preparation to facilitate SFP; 3) experience with SFP and other prevention programs; 4) beliefs about program quality and effectiveness; 5) beliefs about program fidelity and adaptation; and 6) self-reported implementation practices. Several interview questions regarding experience with SFP, experience with other prevention programs, and effectiveness of the curriculum were adapted from Ringwalt and colleagues (2004): “How prepared did you feel to implement the program after going through training?”, “How effective did you think SFP is overall?”, “Do you feel that the program content is an effective curriculum in delivering the program’s message?”, “When were you trained?”, “How comfortable do you feel with the way you have to teach when implementing SFP?”, and “How much do you enjoy delivering the program?”. The remaining items were developed for the current study. Closed questions assessing beliefs about fidelity were rated on seven-point Likert-type scales, and included the following: “How acceptable do you think it is to add material that the written program did not address?”, “How acceptable do you think it is to leave specific material in the written program out?”, “How acceptable do you think it is to change material in the written program?” Open-ended questions were designed to A focused approach to assessing fidelity 11 elicit information about actual implementation practices (e.g. “Can you give some examples of instances in which you added things to the program?”) and reasons for adaptations (e.g. “Was there a particular reason for your additions?”). Questions about implementation practices were prefaced with an introduction intended to minimize social desirability bias, e.g. “We’ve found that many facilitators change things in an effort to better serve their participants. What differences are there between the program as written and the way you deliver it?” Coding of Qualitative Responses from Interviews We used a grounded theory analytical approach to code the interview data (Glaser, 1992; Glaser & Strauss, 1967). Grounded theory analysis is an iterative process in which interview data are examined to identify emergent themes or categories. Those themes are then compared to the data, and in this process finer-grained themes or categories may emerge, which are then used to re-examine the data, and so on until a comprehensive theory (coding scheme) has been developed. Using this process, we identified 13 categories of types of adaptation (additions, changes, or deletions) (see Table 1) and 15 categories of reasons for adaptation (see Table 2). We specified characteristics of each category and determined that the categories were mutually exclusive and exhaustive. A postgraduate project coordinator and graduate research assistant then coded all transcripts, and we computed kappa coefficients to determine interrater reliability for types of adaptations and reasons for them, separately for additions, changes, and deletions. Kappas ranged from 0.86 (reasons for deleting material) to 0.92 (reasons for changes) and averaged 0.89. For each of the 13 types and 15 reasons, reported changes and reasons were assigned either a “1” (for presence, if they reported the type or reason at least once) or “0” (for absence).